Leveraging a rolling cross-section design in panel studies to investigate police legitimacy

Monica Gerber and Cristóbal Moya


London School of Economics and Political Science

May 22, 2023

Today

  1. Motivation
  2. The rolling cross-section design
  3. This Study: a rolling cross-section panel design
  4. Implementation challenges

Motivation

Panel studies

  • Confirmatory research
  • Individuals are interviewed at two or more points in time
  • Possible to make inferences about changes in attitudes or behaviours at the individual level

Motivation

Panel studies

  • Confirmatory research
  • Individuals are interviewed at two or more points in time
  • Possible to make inferences about changes in attitudes or behaviours at the individual level

Rolling cross-sectional design (RCS)

  • Exploratory research
  • Cross-section: Data obtained at a specific point in time
  • RCS: distribute interviewing within a cross-section in a controlled (random) way over time

The rolling cross-section design

Unexpected event during survey design

Unexpected event during survey design (UESD): “research design that exploits the occurrence of an unexpected event during the fieldwork of a public opinion survey to estimate its causal effect on a relevant outcome by comparing responses of the individuals interviewed before the event \(t_i<t_e\) (control group) to those of respondents interviewed after the event \(t_i>t_e\) (treatment group).” (Muñoz et al., 2020, p. 187)

  • Effects of naturally occuring events that cannot be manipulated through controlled experiments (external validity)
  • Assumption that interview time is independent from the time when the event occurs

The rolling cross-section design

The “expected-unexpected” event during survey design

  • Temporal heterogeneity as an opportunity to move closer to causal inference (Johnston & Brady, 2002)
  • Steps:
    • Total sample is divided into multiple replicates (smaller random samples)
    • Replicates are assigned to be interviewed at different intervals of times
  • By itself it is not able to capture individual change

The rolling cross-section design

Assumptions (1)

  1. Temporal ignorability: “for any individual, the potential outcomes must be independent from the moment of the interview” (Muñoz et al., p. 189)
  • Moment of the interview should be as good as random
  • In RCS moment of interview is random
  • Protocol to ensure equal procedures to contact and recontact sample units
  • Reachability: ease of contact may vary between respondents
  • Attrition: nonresponse may be correlated with potential outcomes

The rolling cross-section design

Assumptions (2)

  1. Excludability: “any difference between respondents interviewed before and after the event shall be the sole consequence of the event” (Muñoz et al., p. 189)
  • Timing of the interview \(t\) affects \(Y\) only through event \(T\)
  • Beware of collateral events, simultaneous events and unrelated time trends
  • Need to characterize the event and its reactions

The rolling cross-section design

Assumptions (3)

  1. Compliance: “some of the subjects were not really exposed to the event”Assignment to the treatment group \(t_i>t_e\) was assumed to perfectly correspond to actually receiving treatment \(T=1\).” (Muñoz et al., 2020, p. 194)
  • Need to measure compliance

The rolling cross-section design

Example

This Study

Police legitimacy

  • Police work requires public support to be able to function in a democratic context
  • Beliefs that the police act in procedurally just ways and are legitimate facilitate compliance and cooperation with authorities
  • A socially legitimate police force should require little violence to confront opposition
  • However, the world has seen many cases of violent demonstrations, police violence and human rights violations

This Study

The Chilean case

  • Chile’s main police force, the Carabineros, was among the most trusted institutions (Dammert, 2019)
  • Yet, recent events of corruption and excessive use of force during the social uprising in October 2019 have produced a serious loss of trust
  • Crisis of legitimacy: proposals of “deep reform” and even the dissolution of the institution
  • Need to understand relationship between police and citizens in Chile during the next crucial years

This Study

A rolling cross-section panel design

  • Probability-based online panel of the Chilean population (18-65 years old)
  • n = 5.300 in Contact Survey

Implementation challenges

  • Narrower bandwidths:
    • Will reduce N and statistical power
    • Can reduce bias in the presence of collateral events or confounders correlated with time of the interview
    • Can increase management costs
  • Interview window: for how long should respondents have the chance to complete the survey?